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Declaration of Inventor Yoram Gat 

1, Yoram Gat, hereby aftinn and declare that: 

1 . I am one of the two mventors of the invention provided described in the 
claims of the patent application entitled "Image Segmentation Using Branch and Bound 
Analysis", lilcd March 31, 2004, U.S. Patent Application No: 10/815,89(5 (hereinalier 
referred to as ilie " Application"). Intel Corporation, of Santa Clara, California, is the 
assignee of the Application 

2. Honst Haussccker and I are the co-inventors oi lhe abovc-dcscribcd patent 
application and the co-invcntors of the subject matter described and claimed therein. 1 he 
claims of the Application include contributions of mine and of co-inventor Horsi 
Ilaussecker. 

3. I am the autlior of the article entitled "A Branch-and-Bound Technique for 
Nano-siTucture Image Segmentation", provided in the Proceedings of the 2003 
Conlerencc on Computer Vision and Pattern Recognition Workshop (the ' Aniclc 'i. I hc 
Article is attached hereto as Exhibit A. 

4. The Article describes certain portions of the invention described in the 
claims of the Application that were conceived and developed by me. Other elements of 
such claim.s include portions of the invention conceived and developed by co-inventor 
Hi>rsi Haussccker. 
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Exbibit A - Exliibit A is "A Rranch-and- Bound 1 cchnique for Nuno-siruciure 
Image Segnieniation'\ from the Proceedings of the 2003 Conference on Computer Vision 
and PaUem Recognition Workshop. 

I hereby declare thai all statements made herein of my own knowledge arc true 
and that all statements made on infomiaiion and belief are believed to be true; and ftinlier 
that lhe55e statements were made whh the knowledge that willftil false statements and the 
like are punishable by fine or imprisonment, or both, under Section 1001 of Title 1 8 ol 
the United States Code, and that such willfiil false statements may jeopardize the validity 
ol'ihe above-identified application or an patent issued therefrom. 

Respectfully submitted. 



Date 



., 2008 



Yoram Gal 




Docket No.: 42P17823 
Applicaiion No.. 10/815,8% 



Page 3 oi :\ 



EXHIBIT A 



Docket No.; 42P17823 
Application No.: 10/815,896 
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Abstract 

linages q/ ptano-struawes are ofien noisy. On the other 
hand^ in maojt settm$t there is quite a hi of modei knefwl- 
edgf rr^aatiing the obserxTd snvcfwiex. This paper /to* 
poses a nxthod forstptient^g an utia$e mln$ o geometric 
model of the the observed structure. The resulting segment 
tation is gmronteed to be globaily ^ptma!, for an cqAidily 
specified scorefunction. This property provides a great deoi 
of robustness to the algorithm. 

The aigorithm presented explores c pre-defined space 
of tBesm^ntations using a branch-artd-botrnd aJgoritfmt, // 
tttminaies iftose pans of the space that are prvvQbiy poor 
and explores infUriher detai! the more prtmdstttg pans cf 
the space. 

An ecample <^ a segmentashn thm cm be cbutbied 
in dds woy is a ttrai$hr line segmentation ef an image 
into 2 regions that mlnimizies (he intensiry variadm %itMtt 
the regiont. Results A&^ing extraction of specific nano' 
structures one presented. 

A triviai variation on the atgorithm can find a masmum 
O'posteriori probabfUry estimate cfthe ugntenmtion what 
(here exists an a-pricn distribunon oi*erthe segmentations 
and the objeaive function is imetpretedas the iHte^howtcf 
the image given Uu segjmeKtatiorK 

1 Introchiction 

linage s^sntcniation is a mdety exi^or^d pfobUm in Im* 

ajee proceMung. It is tho operalkm of poftitioning an iou^ 
into regions uttlcb conosfMmd to aseamD^fal objects. In ths 
coQiext of luno-visloa this ctten tovolvcs findl^s («iaUvely 
simple, wcd-ddfined shapes such a» lines, dccles or redan* 
gles m n»sy inu^s. 

One of the oomnon meihocb for sejimemation is region- 
basal scgmeatoUon. Regioa-boscd KgmcnUtion sicthods 
ascmpi ID find io tbc iraage icgioos i»hich oro hooogo* 
neonst aeoMtdiag to some criteiioa. sudi as color, intensity 
or textofo. The ob^ctivo is to find regioos thai aro both 
bonno>^neo«s and POSSESS £ood sbapo pioperties • such as 
oooioctcdoG&s tiod edge snaoothacss. 



Iho rocthod proscobed hiro handles both pans of ths ob> 
jeahre cacpl^iily. The siv^e proper tics of the segaicatotions 
andsr consid^ratios ore d&finod. and among oil scgme&ta- 
ticvis with those pcopcittes, dv) one xkhtdh maxliTHzes an 
explicit honc^ndiy criti?rton isobtained. 

Ihi guamatccd optimal ssgmcmatioo is obcain^j osins 
a brancb-and-bottod (BSlB) s?aicb approach. Such an ap- 
proach has been used hefoce in ima^e analysis for object 
rocogmtion (12], [3]). Discussioa of UiorvlaiioiKlijpof this 
apprccch and other mothods teUOied lo the one proposed Is 
presented in section 2. 

laformatty fho setup is as folk>vbs: A set of possible ss»g- 
mentatioos is being considered. Say. tte set of all straight- 
tine segnocntations of (ho image into two ports. Given a par- 
ticatar segmentation, on objective function gives the cost of 
the scgmentatioQ, uith tower costs preferable. 

It is assumed that a lower bound of ths ooi&t over sets 
of SGgmeatations can be cotculatod. That is, ghen a set of 
sesn»ntatlon&. a value caa be detertnlned which is less than 
or ecp al to the cost of the best (least costly) segmcntatKm in 
the set. While la genefol the Imief bound will not t>c equal 
toih@cost of the best ssgmmaim In the set, b is assunied 
that forsmall seta the bound is tight. 

This setup is formalized in Section 3. 

The 6&B algprithm that dstcrralnes the globally best 
segmentation wodcs as follows: It laaiotains a prbrity 
queue, where the elements hi the queue ore xts of seg* 
iscotMioas and their priority is equal to the lower bound 
of the cost of Ukt best segmentttioa io the set (the smiiUef 
the bound, the higher ths priority). 

TbB set with highest priority In iha queue Is removed, 
and split into parts. Each of those parts is then insetted 
to Ibo quelle. The olgorithm stops when the set wUh the 
highest priority cosiaifls only one segnKotation (or some 
other Stopping criterion b oi^). 

A more formal descri ptioD of the tilgorithm is presented 
io Section 4. 

Section 5 describes interpretations of tt» setitp In lhesta> 
tisticol terms aad Section 6 presents a few examples of the 
way ttie algooSun petfonns with aatml and syntlietic iiiK 
ages. 
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2 Relsitioii to previous work 

The generalized Hou^b transform (HT) (e.g. is a 
SGflcric method for locaUng ponunelerizod faratGes of ob- 
jects in on image, li is veiy similar In approach to Uhs 
method proposed hen in (he sense that bolb hove on ex* 
plicil (epresemation of tte shape that is to be located. The 
objective functioa of the Hough transform is based on votes 
Bocumulotod from pvtelSi where cocb pixel Is seen as voting 
for jome pacamctn' values. In the setap hsfe th? objective 
function is unrestricted In bs fccno. 

A major difrerenee between Ihs method proposed hem 
and HT is that HT is implemented by quantizing (he param- 
eter space and evalaatins the objective foncUon for each 
poim in (he quantized porameter space, while (lie BftB 
nv?t}iod described hero uxnks in cominacms space and uses 
bounds to handle cvgioiks of the paraaieter spaos withoat 
evahiating dense sets of points to those regions. 

The Hou^ transfoncD thcrefote has to strilEe a balaacc 
between quaatizing too oooisdy and iacunring opiMmima- 
(toB errors and quantizing too Hoely and incarring liigh 
oomputtiional oqsl The B<fiB method avoids approzima- 
lioB onm^ ultojether aad lias the potential lo anive q^ickiy 
at an exxt sotuikm when the ot>joctivo fsnctkMi is well be- 
havixi. 

A dilTcrefit approxh to image anal/sls and segaientation 
that is rclated to the one presented hcic is thai of Mount, Ne- 
lanyaba and la Moipie ([h]), fallowing \2] they consider 
th^ problem of matching point using partial Haujdoiff 
distance. They suggest n B&B algorithm to obtain good 
matdie^. Ihs approach htn is very simtbr, but the ob- 
>?ctive function considered here is not restricted to being 
the partial HausdortT distance, aad th; focus is not on poiat 
matching. 

Lastly^ anot}K^r woit (hot can bo coatrasted w^iih the ap- 
proa:h hens is that of Shi and Malik ((5}). As is dc3oe here, 
thsy address the gegmenioiion problem as an optimization 
problem. Rather thm having a pre-ddtned $sn of possible 
cd|«; shapes. Shi and Malik have the stupe criterion as part 
of the objective faoctioa. This allaws for aiuch greater vari* 
ety of shapes of segment edges, but reduces the coatiol the 
user has over desired cci$e shapes. For example, ii wtwld 
be dtlTtcalt to enforce stmighl or circa tar edges, if these are 
dcstmbk due to sooie a^riori considerations (os boHcn the 
case ia the coniez i of nano vision) . 



3 The setup 

The iaiage is implicitly assamod fixed ihroughoot tte <l&* 
cushion. The image data may Include any pn>-p(ooessiag of 
the raw image sach as texture features. 



Let P be the set of pixels ia the itnqgc: 

P«t(i,i):i = l.....n^.i«l IV). 

Emun pl r For UhBtradoii* tfaa geaecal 4eRcdptiooef (ho 
uciip sad the Atgodilun h Moompoaied by a eottCf»t» 
ftsanipift Thft auMnptoisM foliowrK: EachpixelpcP 
u ttJBodattd with a meoSar v(p) • il> int oaniy wbie. 
toy. Stn%bt bocw 2«fntf cegnicDiatuMf an cocndoncd, 
Aoosg at] such fgmfwadon*. the oim wbick mioi* 
loixc* the sua of abaohttD «ofoe draiicQ* of 6e 
IMS u £ram ih« Mgmem ntedLsn iib tenghc 

The dciflBb of the ft ^ flp rithni far thii particrf»j««yawi- 
tadoa ptiobleta ut pw<qated utrngnKlft the yaeoc^lfr 
cciiptioii of Aee^^ofithn* 



tel 5 be a parameteiized fimiily of /T^part segmeata* 
tions. Forcxoji^plti the set of aB straight lino isa foimilyof 
2-pan segmentations. !tmnally. any a € <5 is amap fiom P 
ia to the set of cell indices {!,..., K\, 



E xu mp t e tht space of tmiisbt Um cegmeotoacni cva 




Let I bo (he 5e,gmeotatkm loss fanctioo, scormg each scg* 
meotoiioii ocoardio^ to soms (fixed) criterion: 

l :S^ R. 

Ri-amplrTbo CTiirnple «€ tf mc u liilM» loie tnue6ou b 
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wbsio 

wikb Vjt^t tMu$ the medun value of Um> set {«^) : 

JUe al^tbm described here is aimed at flAding a sey- 
niertaUon that imrainize» the objective funaioo I enter tho 
xt of ^gmentattons 5. D assuixi?s tbo existeoce of lbs fol- 

towing: 

• A a^finifxg faocUon /? - 

This ftinciioj) bn>aks up scu of scgmcniaiioiu iato 
smaller sets by impplft^ a set of scgmcntmiom SCS 
Into a set of eegnentation sets 5|, . . . , 5) w-hicb fonn 
a paititioD of 5. The furciioa needs only be definsd 
for ihsx sets which can be obtained by refining 5. 

Gmnple For ootacioiukl ease define 
S'^M»ln>.'t) = {'V : € (^AI,r € (ro, n])r 

Th» refinat^ ftmedaa ttr lbs raiawiflft atnip U 
dcfioed oo* mapf tp tcti oC aet oCicpiKnlii* 
li«a» of the form 5)i^^t{» tn»,rs|. 
It u defined as (oSUwv. 

wha» ^ o (9o ^t)/3 and r' 8 (no + n)/!. 




• A poftial se^nienlatioii function Af - 

Hie fUKtion M must have the foUowtng propcity: For 
anyp € V, 



That is. ihe function Af pcovides a sopenet of oD the 
cells thai a pixel may be mapped to by a id of seg- 
(acniatioos. As is tho coso «itb tha icfiniofl fimctioa 
R, the Amnion M needs only to be defined wben the 
fint ai;g^ineat is asuteet of S thai can be obtained by 

Kwimtile A (sftiel iggmmtariop f oftcdoo for 
9at0i.»b,riaadlflt 
fo B miD ) eos« -i- 0 ^) ahi 9 : 



Let 



(I) if*»<ni. 

if«o>n, 
{1,2} othmnw 











+ 











) A (SGUtial ae^nealation loss function B • 

This fnnction maps a peutiai segmentation of die pix- 
els is the iou^ to Q loss vaJue. This loss vahie most 
be less than or equal to oil loss >4iliies ossociatod with 
any augmentation that is consish^it with ihs partial sc^- 
mentaiioa. That is if T is a map from pixels to si^ds 
of{l,...,K}.thcn 

B{T) < inl {({$) : a e for aU p € 7^. tf (p) € T{p)) . 



ProoBodlhas dfthp 2003 Corritaranop cn OoopMicr Vlskn and Raitoro thooatiSan Wortehop(CVPftWtl9) 
10634919103$ 17j00O3003CEE 



Docket No.: 42P17823 
Application No.: 10/815,896 



Exmnple A partinl MsncMitiott losi fuRctioo 
Cor *e fmctai of tb» oumpla IS 

wbem 

wilfa vr,i being tho a«diao miJiw of kb« set 



4 A branch-and'bound algorithm 

Tlio bmncb'Ond'bcund algorithm which is used to mmh 
for an ofHimal scgmcntatioa Is a standanl A* procodun) 
((4]). Thi3 procedure usasi a priorit)- qucuo to store 'stales'. 
A slate amy be a terminal state uliicb means that vstten it 
reaches the t»2)d of the queue (ho search has terminated. 
Otherwise the state is mivtsnnina). and so t»tico it teaches 
ths Used of the quetie. it is removed from the queue and all 
iIk states thatars immediaiely teachablci (torn thai state at? 
imencd to the queue. Each sttue in the queue hns a priority 
scoiB associated witb it and die stale with the louiest pri* 
ority score (t.e„ highest pnoiily) is coosidefcd to be at the 
head of the queue. 

In the pnsscat setup th? states are sohsaets of tt» seamen- 
lation space S. For any ootHcnninaJ staKi. 5. the states 
uhid) are inunedi ately r^achahle fiom S aie ^vseo by R(S). 

Vtin$ and J?, the priority function t> 

ts defined as follows: 

L(S) « B(T), 

vkUcfo the p;iftiaJ segsicDtation T is ghren by T0>) =: 
^H^fP)' By constniction. tho functbtk L provides s ktwor 
bound for the toss of oriy fiegntenintton «hich is on element 
of the set whicli is its ariomeot: 

lr(5)<lnf{f(«):«€5}. 

the tighuvss of os a bouad to inf{/(«) : s ^ S) 
is a major factor in determining the efltdeocy of the seg- 
nvntaiion algorithm. For the algorithm to converse to ihs 
optiouil segfDcniation it is necessary thai the follcmtig limit 
holds for any se<|ueoco of sets Sb. ^i. * . generated by re* 
fining 5: 

.HmL(5i)«/(nSo5f). 



A slate is terminal if aO the se^sniGiitatioos in tbo scig- 
tneotatioa sot q^roe oo all (Pixels, or if fon» oth?r tcnnina- 
ttoo oooditioin is ovt Ths searcb is ijiiliafizcd by inserting 
10 thd qtieae Iho itoto utich cotresponds lotlio wbolesc^- 
menbakm space S, The algorithm b theiefons 

• InitlaUzation Start with eiqply pffexity queue. Inseit 
into the queue the staled, with the ooms^ponding pii- 
orily score L{Sy 

• Extract Extract ths at the tead of the queue - 5. 
RcQtovo it from the queue. If it is a tcnnhml state • 
5io(x outpot 5 05 the solatloD to the optltniioUon piob- 

• bisert Inseit to the ci^eoe ttk> membcfs of ibe set 
B(S) ea with Sio priority sooies 

£r(5j} HSt,) correspondingly. 

• Iterate Go bock to step Extract. 

5 Frequentist and Bayesian sladstl* 
cal Interpretation 

ir the loss function can be infi^rpietcd as the negative 
of the log-likelihood of the image givtrn tiic scgoKntation 
4, then tbs search for minimum loss can be interpreted as 
maximum hicdihood estimation 

Trivially, the sane algorithm can be used to obtaia a 
Bayesian maximum a-postcriori probability (Kf AP) esti- 
mate of the scgmeototion. The loss associated «11h a seg- 
mentatioo «iouki Uien b@ taterpfetod as the sum of the aeg* 
ative of the log-piobabtlity of tha iis^ge given the seginen> 
lotion and the negative of the log tbs a^piiad iirob^ 
of the segmentation. 

In sKklition to obtaining the MAP estlmato, a oedible 
region can bo obtaiood. by cmployti^ a ttmilar algorithm 
to the one prcseoted. This is a r^ioo of the segmentation 
space thai a supports a prosciibed part of ths probability in 
the o-posteriori distribution of th? segmenioiioa. 

In oakr to do this, upper bounds on ihs loss functioa as 
wdl as lower bounds, mod to bo evalaatod 

6 Results for test images 

This seciioa shows the output of ibe segmentation algD> 
rithm when used to exUwt fcatuios f n^m a focused ion beam 
imaging (FIB) cool image. The femuies on copper wires on 
a badi;giDond of silicon. The segmentation space consid- 
ered for each Image was ooasintcted spccUkolly to extroi:! 
Ute expected stntctuitd: on L shaped ooriKf. a T^juaaion or 
a wim terminatioiL 
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"Hiblo 1: Expcfiroent dctmte. AU cxp»imcnts wcic run oa a 
2,4CHz Pi^ntium 4 desktop sysisoL, with 2GB RAM. lau^ 
il7€!i vary Iwiueen 172 x 115 and 352 x 3 O (iixe\%. 



Iho bb}»cUv9 function Tor all Ulc taU cases wos: 

uhero I emimcratcs ttie diflerem oetU^ oixl f» is tbo OKdion 
Intensity vnltie of ttie pixels in cet! L 

Ihe paOisl segnvntation loss foocitoo B nsfxl was as 
followji: 

Bir) = mio (B'(T, oi) + B*(r, oa) + BM(t, qi, oa)) , 
where 

and 

ITtt^ funcUpQ cm be evaiaated in Ume tlud Is imifpntiMUl 
to th; oamberof pos^blo valiies of color intensity, le.. 256. 
The results arc in Figure I . 

Thble 1 shows jjorae details rc^nrding tho cxp^rimsmi. 

lu ileamuaiuute th^ tubustJivsK uftUt? lUgustilmi tu turn!, 
it utis also run oti a &yaih(^tic imago that comoiiuxl an od£& 
obscured by noise. The olgor iihrn is able to nxovier the &is^ 
(Figure 2) ew*n when the noise is high enough to <dia|tef^ 
til? human eye. 

For the ^yuthctic color liooge. the objective function used 
vi^is tho sum of the throe objective values obtain by evalu- 
ating the ^glc chaaoel objsciKe fancUon for each of tb9 
tltfoe color channels. 

7 Conclusions and futiii^ work 

Tho^fcXMk presented licred»moosifatcd thai scgtneniatton of 
naito>structure imogies can be obtained by optimizing asim- 
pie objoctive fa action wcr a space of partitions with pre- 
defined ihape pn>pcfties. Wtb computing power inroitablo 




Figuro I: MffMNtsnano-ftnickiifBdstifineolodcNitcirFIB im* 

ages 
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Dbtaooe. lEEEcfflnsflc Ams.do ptffon aiia(>:s/5aiid im- ^ 

|3J M(ianlDA!..NetanyQtiuN^.afidUMo^^ 

GrBdent A^odUuns for Rolmst Feature Matchfii& Pax- 
tetD raccsgidtjoo 32 ( 13). 

(4] Russsl. S. I . cud Kofvijs. P. (199^ Art/ficiil iMdl/- 
gSB^oce^ Prcnike HaB. 

|5] Shi i. .and MaEt X (3000). rOormlixed cols and im- 
sogmcfllQikui. IEEE craosaitim oo |urttm a 



Bgune 2: (Ibp to bottom) Gcosjid tnilb ed^ ground muti 
•f noise o bpuL output 

today; nndi^g soluttoos tbat ore ; global ly opdnml fe:£siblo 
bat cornioa be dono ID real iim?. 

A useful externum of the algoritbni thai fuxtsea&^d 
would be ixsiixg a siniilaf appftxich to obtain oiaMple local 
optima, in addition to Ihe^IotoQl opUmuni* Tbis would en- 
aUepickiagout fram an ima^e mttUipteobjcdt .with similar 
dupo dumcicriaics. 

Anothsr interested Use of lesearcta wouki be aslng 
ptO^sks based oiyoctivo fufKrUons msteod of the henrbtk: 
functions used in tho present papix. 
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